{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dxFGlX1S53mg"
   },
   "source": [
    "# Ungraded Lab: Training a Deep Neural Network with Time Series Data\n",
    "\n",
    "In this lab, you will build upon the previous exercise and add more dense layers to your network. You will also look at a technique to tune the model's learning rate to make the weights converge faster. This is a useful tip so you can avoid guessing the learning rate before training.\n",
    "\n",
    "The initial steps will be identical to the previous lab so you can run the next cells until the `Build the Model` section. That's where the discussions begin."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w5S4XCUZ6GH7"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BOjujz601HcS"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tjC8TSkp6IiH"
   },
   "source": [
    "## Utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Zswl7jRtGzkk"
   },
   "outputs": [],
   "source": [
    "def plot_series(time, series, format=\"-\", start=0, end=None):\n",
    "    \"\"\"\n",
    "    Visualizes time series data\n",
    "\n",
    "    Args:\n",
    "      time (array of int) - contains the time steps\n",
    "      series (array of int) - contains the measurements for each time step\n",
    "      format - line style when plotting the graph\n",
    "      label - tag for the line\n",
    "      start - first time step to plot\n",
    "      end - last time step to plot\n",
    "    \"\"\"\n",
    "\n",
    "    # Setup dimensions of the graph figure\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    \n",
    "    if type(series) is tuple:\n",
    "\n",
    "      for series_num in series:\n",
    "        # Plot the time series data\n",
    "        plt.plot(time[start:end], series_num[start:end], format)\n",
    "\n",
    "    else:\n",
    "      # Plot the time series data\n",
    "      plt.plot(time[start:end], series[start:end], format)\n",
    "\n",
    "    # Label the x-axis\n",
    "    plt.xlabel(\"Time\")\n",
    "\n",
    "    # Label the y-axis\n",
    "    plt.ylabel(\"Value\")\n",
    "\n",
    "    # Overlay a grid on the graph\n",
    "    plt.grid(True)\n",
    "\n",
    "    # Draw the graph on screen\n",
    "    plt.show()\n",
    "\n",
    "def trend(time, slope=0):\n",
    "    \"\"\"\n",
    "    Generates synthetic data that follows a straight line given a slope value.\n",
    "\n",
    "    Args:\n",
    "      time (array of int) - contains the time steps\n",
    "      slope (float) - determines the direction and steepness of the line\n",
    "\n",
    "    Returns:\n",
    "      series (array of float) - measurements that follow a straight line\n",
    "    \"\"\"\n",
    "\n",
    "    # Compute the linear series given the slope\n",
    "    series = slope * time\n",
    "\n",
    "    return series\n",
    "\n",
    "def seasonal_pattern(season_time):\n",
    "    \"\"\"\n",
    "    Just an arbitrary pattern, you can change it if you wish\n",
    "    \n",
    "    Args:\n",
    "      season_time (array of float) - contains the measurements per time step\n",
    "\n",
    "    Returns:\n",
    "      data_pattern (array of float) -  contains revised measurement values according \n",
    "                                  to the defined pattern\n",
    "    \"\"\"\n",
    "\n",
    "    # Generate the values using an arbitrary pattern\n",
    "    data_pattern = np.where(season_time < 0.4,\n",
    "                    np.cos(season_time * 2 * np.pi),\n",
    "                    1 / np.exp(3 * season_time))\n",
    "    \n",
    "    return data_pattern\n",
    "\n",
    "def seasonality(time, period, amplitude=1, phase=0):\n",
    "    \"\"\"\n",
    "    Repeats the same pattern at each period\n",
    "\n",
    "    Args:\n",
    "      time (array of int) - contains the time steps\n",
    "      period (int) - number of time steps before the pattern repeats\n",
    "      amplitude (int) - peak measured value in a period\n",
    "      phase (int) - number of time steps to shift the measured values\n",
    "\n",
    "    Returns:\n",
    "      data_pattern (array of float) - seasonal data scaled by the defined amplitude\n",
    "    \"\"\"\n",
    "    \n",
    "    # Define the measured values per period\n",
    "    season_time = ((time + phase) % period) / period\n",
    "\n",
    "    # Generates the seasonal data scaled by the defined amplitude\n",
    "    data_pattern = amplitude * seasonal_pattern(season_time)\n",
    "\n",
    "    return data_pattern\n",
    "\n",
    "def noise(time, noise_level=1, seed=None):\n",
    "    \"\"\"Generates a normally distributed noisy signal\n",
    "\n",
    "    Args:\n",
    "      time (array of int) - contains the time steps\n",
    "      noise_level (float) - scaling factor for the generated signal\n",
    "      seed (int) - number generator seed for repeatability\n",
    "\n",
    "    Returns:\n",
    "      noise (array of float) - the noisy signal\n",
    "    \"\"\"\n",
    "\n",
    "    # Initialize the random number generator\n",
    "    rnd = np.random.RandomState(seed)\n",
    "\n",
    "    # Generate a random number for each time step and scale by the noise level\n",
    "    noise = rnd.randn(len(time)) * noise_level\n",
    "    \n",
    "    return noise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z1CL6Bh86TS1"
   },
   "source": [
    "## Generate the Synthetic Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5MFtI3X1FErU"
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "time = np.arange(4 * 365 + 1, dtype=\"float32\")\n",
    "baseline = 10\n",
    "amplitude = 40\n",
    "slope = 0.05\n",
    "noise_level = 5\n",
    "\n",
    "# Create the series\n",
    "series = baseline + trend(time, slope) + seasonality(time, period=365, amplitude=amplitude)\n",
    "\n",
    "# Update with noise\n",
    "series += noise(time, noise_level, seed=42)\n",
    "\n",
    "# Plot the results\n",
    "plot_series(time, series)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t2YvxWcI6lnB"
   },
   "source": [
    "## Split the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Db9kkYnOFKYr"
   },
   "outputs": [],
   "source": [
    "# Define the split time\n",
    "split_time = 1000\n",
    "\n",
    "# Get the train set \n",
    "time_train = time[:split_time]\n",
    "x_train = series[:split_time]\n",
    "\n",
    "# Get the validation set\n",
    "time_valid = time[split_time:]\n",
    "x_valid = series[split_time:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XH-0FgXN6zHA"
   },
   "source": [
    "## Prepare Features and Labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AyBwNWO1FVTS"
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "window_size = 20\n",
    "batch_size = 32\n",
    "shuffle_buffer_size = 1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4sTTIOCbyShY"
   },
   "outputs": [],
   "source": [
    "def windowed_dataset(series, window_size, batch_size, shuffle_buffer):\n",
    "    \"\"\"Generates dataset windows\n",
    "\n",
    "    Args:\n",
    "      series (array of float) - contains the values of the time series\n",
    "      window_size (int) - the number of time steps to average\n",
    "      batch_size (int) - the batch size\n",
    "      shuffle_buffer(int) - buffer size to use for the shuffle method\n",
    "\n",
    "    Returns:\n",
    "      dataset (TF Dataset) - TF Dataset containing time windows\n",
    "    \"\"\"\n",
    "  \n",
    "    # Generate a TF Dataset from the series values\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(series)\n",
    "    \n",
    "    # Window the data but only take those with the specified size\n",
    "    dataset = dataset.window(window_size + 1, shift=1, drop_remainder=True)\n",
    "    \n",
    "    # Flatten the windows by putting its elements in a single batch\n",
    "    dataset = dataset.flat_map(lambda window: window.batch(window_size + 1))\n",
    "\n",
    "    # Create tuples with features and labels \n",
    "    dataset = dataset.map(lambda window: (window[:-1], window[-1]))\n",
    "\n",
    "    # Shuffle the windows\n",
    "    dataset = dataset.shuffle(shuffle_buffer)\n",
    "    \n",
    "    # Create batches of windows\n",
    "    dataset = dataset.batch(batch_size)\n",
    "    \n",
    "    # Optimize the dataset for training\n",
    "    dataset = dataset.cache().prefetch(1)\n",
    "    \n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AdNN0s9qFnTg"
   },
   "outputs": [],
   "source": [
    "# Generate the dataset windows\n",
    "dataset = windowed_dataset(x_train, window_size, batch_size, shuffle_buffer_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ths8EcjQ7CG3"
   },
   "source": [
    "## Build the Model\n",
    "\n",
    "You will use three dense layers in this exercise as shown below. As expected, the number of trainable parameters will increase and the model summary shows that it is more than tenfold of the previous lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TW-vT7eLYAdb"
   },
   "outputs": [],
   "source": [
    "# Build the model\n",
    "model_baseline = tf.keras.models.Sequential([\n",
    "    tf.keras.Input(shape=(window_size,)),\n",
    "    tf.keras.layers.Dense(10, activation=\"relu\"), \n",
    "    tf.keras.layers.Dense(10, activation=\"relu\"), \n",
    "    tf.keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "# Print the model summary\n",
    "model_baseline.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4xWDKTGR7XQd"
   },
   "source": [
    "## Train the Model\n",
    "\n",
    "You will then compile and train the model using the same settings as before. Observe how the loss is decreasing because you will revisit it later in this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "unwiKhWn7nqS"
   },
   "outputs": [],
   "source": [
    "# Set the training parameters\n",
    "model_baseline.compile(loss=\"mse\", optimizer=tf.keras.optimizers.SGD(learning_rate=1e-6, momentum=0.9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "zqVlvuneFqJz",
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "model_baseline.fit(dataset,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "0-5QVzo1Tv-_",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "You can then get some predictions and visualize it as before. Since the network is deeper, the predictions might go slower so you may want to minimize unnecessary computations. \n",
    "\n",
    "In the previous lab, you might remember the model generating predictions using the entire `series` data. That resulted in 1,441 points in the `forecast` list then you sliced the 461 points that aligns with the validation set using `forecast = forecast[split_time - window_size:]`. \n",
    "\n",
    "You can make this process faster by just generating 461 points right from the start. That way, you don't waste time predicting points that will just be thrown away later. The code below will do just that. It will just get the points needed from the original `series` before calling the `predict()` method. With that, all predictions will align with the validation set already and the for-loop will run for only 461 times instead of 1,441. \n",
    "\n",
    "In the next lab, you'll see an even faster way to generate these predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "efhco2rYyIFF",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialize a list\n",
    "forecast = []\n",
    "\n",
    "# Reduce the original series\n",
    "forecast_series = series[split_time - window_size:]\n",
    "\n",
    "# Use the model to predict data points per window size\n",
    "for time in range(len(forecast_series) - window_size):\n",
    "  forecast.append(model_baseline.predict(forecast_series[time:time + window_size][np.newaxis], verbose=0))\n",
    "\n",
    "# Convert to a numpy array and drop single dimensional axes\n",
    "results = np.array(forecast).squeeze()\n",
    "\n",
    "# Plot the results\n",
    "plot_series(time_valid, (x_valid, results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "Lk5VkqNjh8Ws",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "You can then get the MSE and MAE for reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "-kT6j186YO6K",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Compute the metrics\n",
    "print(tf.keras.metrics.mse(x_valid, results).numpy())\n",
    "print(tf.keras.metrics.mae(x_valid, results).numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "tQB2gyq18YvD",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Tune the learning rate\n",
    "\n",
    "You saw that the training went well with the initial learning rate that you chose (i.e. `1e-6`). However, you're not yet sure if it is the best setting for this particular model. It might seem inconsequential in this simple model but when you have more complex ones, spending some time to tune the learning rate can lead to better training results. You will see how to do that in this section.\n",
    "\n",
    "First, you will build the same model architecture you just used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "tnCe_nBKu7RB",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Build the Model\n",
    "model_tune = tf.keras.models.Sequential([\n",
    "    tf.keras.Input(shape=(window_size,)),\n",
    "    tf.keras.layers.Dense(10, activation=\"relu\"), \n",
    "    tf.keras.layers.Dense(10, activation=\"relu\"), \n",
    "    tf.keras.layers.Dense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "CTSWS74Qj-G0",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Next, you will declare a [learning rate scheduler](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LearningRateScheduler) callback. This will allow you to dynamically set the learning rate based on the epoch number during training. As shown below, you will pass a lambda function to declare the value of the learning rate. It will start at `1e-8` at epoch 0 and is scaled by `10**(epoch / 20)` as the training goes on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "6FDVah6-885n",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set the learning rate scheduler\n",
    "lr_schedule = tf.keras.callbacks.LearningRateScheduler(\n",
    "    lambda epoch: 1e-8 * 10**(epoch / 20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "M4SJZxL1lqPX",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "You will then compile the model. Just to note a subtle difference with the lecture video, you don't have to set the `learning_rate` argument of the optimizer here before compiling. You can just leave the default (i.e. [`0.01` for SGD](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/SGD#args)) and allow the learning rate scheduler to set it dynamically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "66MDQGRc9AKU",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialize the optimizer\n",
    "optimizer = tf.keras.optimizers.SGD(momentum=0.9)\n",
    "\n",
    "# Set the training parameters\n",
    "model_tune.compile(loss=\"mse\", optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "zMgbgBYzmXBi",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "You will pass in the `lr_schedule` callback in the `callbacks` parameter of the [`fit()`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit) method. As you run the training below, you will see the learning rate at a particular epoch denoted by `lr` in the console output. Notice that it is increasing as expected based on the lambda function you used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "Ef5o5DmQkuuy",
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "history = model_tune.fit(dataset, epochs=100, callbacks=[lr_schedule])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "Sz9yLyRkkuet",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Next step is to plot the results of the training. You will visualize the loss at each value of the learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "2ZaNsM2IgCd_",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the learning rate array\n",
    "lrs = 1e-8 * (10 ** (np.arange(100) / 20))\n",
    "\n",
    "# Set the figure size\n",
    "plt.figure(figsize=(10, 6))\n",
    "\n",
    "# Set the grid\n",
    "plt.grid(True)\n",
    "\n",
    "# Plot the loss in log scale\n",
    "plt.semilogx(lrs, history.history[\"loss\"])\n",
    "\n",
    "# Increase the tickmarks size\n",
    "plt.tick_params('both', length=10, width=1, which='both')\n",
    "\n",
    "# Set the plot boundaries\n",
    "plt.axis([1e-8, 1e-3, 0, 300])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "obCq2g0qn0GD",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "The generated graph above shows the values of the range of learning rates that leads to lower losses (i.e. sloping downward) and also which ones cause the training to become unstable (i.e. jagged edges and pointing upwards). In general, you will want to pick a point in a downward slope. That means the network is still learning at that point and is stable. Choosing close to the minimum point of the graph will make the training converge to that loss value quicker, as will be shown in the next cells.\n",
    "\n",
    "First, you will initialize the same model architecture again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "F7vzeS6BKUb_",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Build the model\n",
    "model_tune = tf.keras.models.Sequential([\n",
    "    tf.keras.Input(shape=(window_size,)),\n",
    "    tf.keras.layers.Dense(10, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(10, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "9gjrVfgWp3ku",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "You will then set the optimizer with a learning rate close to the minimum. It is set to `4e-6` initially but feel free to change based on your results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "nVHuxnXMKjja",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set the optimizer with the tuned learning rate\n",
    "optimizer = tf.keras.optimizers.SGD(learning_rate=4e-6, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "pyrhSDU5qPn6",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "You can then compile and train the model as before. Observe the loss values and compare it to the output of the baseline model you had before. Most likely, you will have met the final loss value of the `model_baseline` within the first 50 epochs of training this `model_tune`. You will also likely have a lower loss after all 100 epochs are done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "rDNeYqqzqQ-s",
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set the training parameters\n",
    "model_tune.compile(loss=\"mse\", optimizer=optimizer)\n",
    "\n",
    "# Train the model\n",
    "history = model_tune.fit(dataset, epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "8W-zX3LCrZc2",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "You can plot the `loss` values by getting it from the [History](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/History) object returned by the `fit()` method. As you can see, the model is still trending downward after the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "iXBMO1HM9AHX",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Plot the loss\n",
    "loss = history.history['loss']\n",
    "epochs = range(len(loss))\n",
    "plt.plot(epochs, loss, 'b', label='Training Loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "xakiRU7R7WAo",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Plot all but the first 10\n",
    "loss = history.history['loss']\n",
    "epochs = range(10, len(loss))\n",
    "plot_loss = loss[10:]\n",
    "plt.plot(epochs, plot_loss, 'b', label='Training Loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "XxDbVYFusI3O",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "You can get the predictions again and overlay it on the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "mIi2fTzPTJxe",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialize a list\n",
    "forecast = []\n",
    "\n",
    "# Reduce the original series\n",
    "forecast_series = series[split_time - window_size:]\n",
    "\n",
    "# Use the model to predict data points per window size\n",
    "for time in range(len(forecast_series) - window_size):\n",
    "  forecast.append(model_tune.predict(forecast_series[time:time + window_size][np.newaxis], verbose=0))\n",
    "\n",
    "# Convert to a numpy array and drop single dimensional axes\n",
    "results = np.array(forecast).squeeze()\n",
    "\n",
    "# Plot the results\n",
    "plot_series(time_valid, (x_valid, results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "KExPe7qJsTxk",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Finally, you can compute the metrics and you should arrive at similar figures compared to the baseline. If it is much worse, then the model might have overfitted and you can use techniques you know to avoid it (e.g. adding dropout)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "id": "W-GPjL2wv0yc",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(tf.keras.metrics.mse(x_valid, results).numpy())\n",
    "print(tf.keras.metrics.mae(x_valid, results).numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "U9ViZ1Mcs11C",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Wrap Up\n",
    "\n",
    "This concludes the exercise on using a deep neural network for forecasting. Along the way, you did some hyperparameter tuning, particularly on the learning rate. You will be using this technique as well in the next labs. Next week, you will be using recurrent neural networks to build your forecasting model. See you there and keep it up!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "C4_W2_Lab_3_deep_NN.ipynb",
   "private_outputs": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
